Increasing the speed of computation of a volumetric scattering render technique

ABSTRACT

Presented here is a system and method to increase the speed of computation of a volumetric scattering render technique. The volumetric scattering can include path tracing which simulates interactions between a virtual ray of light and a volume. The interaction can include reflection of the virtual ray of light of a particle within the volume. The system can obtain a threshold number of interactions between a virtual ray of light and a three-dimensional object through which the virtual ray of light is traveling. As the system performs the simulation, the system can compare a number of the interactions to the threshold number. Upon determining that the number of interactions is equal to or exceeds the threshold number, the system can terminate the simulation and approximate interactions between the virtual ray of light and the volume using a second rendering technique that is computationally less expensive than simulating the interactions.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. Utility patent applicationSer. No. 17/482,310 filed Sep. 22, 2021 which is a continuation of U.S.Utility patent application Ser. No. 17/355,082, filed Jun. 22, 2021,which claims priority to and the benefit of the U.S. Provisional PatentApplication Ser. No. 63/210,926 filed Jun. 15, 2021 all of which areincorporated herein by reference in their entirety.

BACKGROUND

Subsurface scattering (SSS), also known as subsurface light transport(SSLT), is a mechanism of light transport in which light that penetratesthe surface of a translucent object is scattered by interacting with thematerial and exits the surface at a different point. The light willgenerally penetrate the surface and be reflected a number of times atirregular angles inside the material before passing back out of thematerial at a different angle than it would have had it had beenreflected directly off the surface. Subsurface scattering is importantfor realistic 3D computer graphics, where it is necessary for therendering of materials such as marble, skin, leaves, wax, clouds, milk,etc. If subsurface scattering is not implemented, the material may lookunnatural, like plastic or metal. One drawback of subsurface scatteringis the computational cost required to compute it.

SUMMARY

Presented here is a system and method to increase the speed ofcomputation of a volumetric scattering render technique. The volumetricscattering render technique can include path tracing that simulatesinteractions between a virtual ray of light and a three-dimensionalobject such as a volume. The three-dimensional object can be smoke,cloud, marble, skin, milk, etc. The interaction can include reflectionof the virtual ray of light of a particle within the volume.

The system can obtain a threshold number of interactions between avirtual ray of light and a three-dimensional object through which thevirtual ray of light is traveling. As the system performs thesimulation, the system can compare a number of the simulatedinteractions to the threshold number. Upon determining that the numberof simulated interactions is equal to or exceeds the threshold number,the system can terminate the simulation and approximate interactionsbetween the virtual ray of light and the three-dimensional object usinga second rendering technique that is computationally less expensive thansimulating the interactions. The second rendering technique can includea closed form solution, a dipole approximation, a sum of Gaussiansapproximation, etc.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

Detailed descriptions of implementations of the present invention willbe described and explained through the use of the accompanying drawings.

FIG. 1 is a block diagram that illustrates a wireless communicationssystem.

FIG. 2 shows multiple rays generated per pixel.

FIGS. 3A-3B show methods to increase a speed of computation of avolumetric scattering render technique.

FIG. 4 shows a system to present the images with volumetric scatteringto a user.

FIGS. 5A-5C show images obtained using various combinations of thesimulation and cheaper rendering techniques.

FIGS. 6A-6C show areas of an image that are rendered using thesimulation and areas of the images that are rendered using a cheaperrendering technique.

FIG. 7 is a flowchart of a method to increase a speed of computation ofa volumetric scattering render technique, according to one embodiment.

FIG. 8 is a flowchart of a method to increase a speed of computation ofa volumetric scattering render technique, according to anotherembodiment.

FIG. 9 illustrates an example visual content generation system as mightbe used to generate imagery in the form of still images and/or videosequences of images.

FIG. 10 is a block diagram that illustrates a computer system upon whichthe computer systems of the systems described herein and/or visualcontent generation system may be implemented.

The technologies described herein will become more apparent to thoseskilled in the art from studying the Detailed Description in conjunctionwith the drawings. Embodiments or implementations describing aspects ofthe invention are illustrated by way of example, and the same referencescan indicate similar elements. While the drawings depict variousimplementations for the purpose of illustration, those skilled in theart will recognize that alternative implementations can be employedwithout departing from the principles of the present technologies.Accordingly, while specific implementations are shown in the drawings,the technology is amenable to various modifications.

DETAILED DESCRIPTION

The description and associated drawings are illustrative examples andare not to be construed as limiting. This disclosure provides certaindetails for a thorough understanding and enabling description of theseexamples. One skilled in the relevant technology will understand,however, that the invention can be practiced without many of thesedetails. Likewise, one skilled in the relevant technology willunderstand that the invention can include well-known structures orfeatures that are not shown or described in detail, to avoidunnecessarily obscuring the descriptions of examples.

Increasing the Speed of Computation of a Volumetric Scattering RenderTechnique

The system presented here combines an expensive-and-accurate volumetricscattering rendering technique, namely, simulating interactions of avirtual ray and particles inside the volume and a less computationallyexpensive technique that approximates the results obtained from thesimulation. The simulation can utilize path tracing. Path tracing tracesa virtual ray of light (“ray”) coming from a light source or a camerathrough a volume. The ray can include one or more wavelengths ofelectromagnetic radiation. The ray is refracted upon entering the volumeand proceeds to travel through the volume interacting with the volumeparticles. The path tracing technique simulates the interactions betweenthe ray and the volume, such as reflection, scattering, and absorption,that generate a random path that the ray takes traveling through thevolume.

To reduce the amount of computation in path tracing, the user canspecify a threshold indicating a certain number of reflections of theray within the volume, after which the path tracing computation stops.If the ray exits the surface prior to crossing the threshold, thecontribution of the ray to the surface color is computed. If the raydoes not exit the surface prior to crossing the threshold, the tracingof the ray within the volume is terminated, and the contribution of theray to the surface color is calculated using the less computationallyexpensive technique. The less computationally expensive technique can bea closed form solution, a dipole approximation, and/or a sum ofGaussians approximation, etc.

FIG. 1 shows multiple virtual rays traveling through a volume. Thecamera 110 creates an image of a scene 100 including a three-dimensionalobject 120 and a light source 130. The three-dimensional object(“object” or “volume”) 120 represents a volume through which lightcoming from the light source 130 can scatter, such as marble, skin,leaves, wax, clouds, fog, milk, etc. A virtual ray 140 represents alight ray scattering through the object 120. In materials that scatterlight, the contribution of scattered light significantly contributes tothe quality of the final image.

Volumetric scattering is a rendering technique that attempts to simulatethe scattering of light between the light source 130 and the camera 110.Path tracing is a volumetric scattering technique. Path tracing emits avirtual ray of light 140 from the camera 110, scatters the ray throughthe volume 120 at multiple points 150, 160 (only two labeled forbrevity), and traces the ray back to the light source 130. A scatteringevent includes the virtual ray of light 140 reflecting from a particlein the volume 120. A ray that does not reach the light source 130, suchas ray 170, contributes no color to the final pixel in the image and isignored.

To generate a high-quality image, a rendering system (“renderer”)randomly generates millions of rays per pixel, such as ray 140, perframe. Each ray 140 can scatter multiple times, such as hundreds oftimes, while traveling through the volume 120. Simulating hundreds ofscattering events for millions of pixels millions of times becomescomputationally expensive. Further, as the number of scattering events150, 160 increases, the contribution of the ray 140 to the final pixelcolor decreases.

FIG. 2 shows multiple rays generated per pixel. The camera 200 recordsan image 210 of the object 220. The image 210 contains multiple pixels230, 240 (only two labeled for brevity). To create the image 210, arenderer generates multiple rays 250, 260 (only two labeled for brevity)per pixel 230, 240. Even though the full set of rays 250, 260 do notcover the area of the pixel 230, the renderer generates millions ofrays, and the color computed for each of the millions of raysapproximates the true pixel 230 color.

When recording a sequence of images such as for a movie or video, therenderer generates independent and different rays for subsequent images,that is, frames, in the sequence. That is, the rays for frame N can beas shown in FIG. 2 , while the rays for frame N+1 go through differentpoints in the pixel 230 than the rays 250, 260. The random variation ofrays from frame to frame enables generating aesthetically pleasingimages. By contrast, if the same rays were used for all the frames, theviewer of the movie or the video would be under the impression that themovie was recorded through a shower curtain. The rays generated betweendifferent frames can be generated according to the Monte Carlo samplingprinciple.

FIGS. 3A-3B show methods to increase a speed of computation of avolumetric scattering render technique. As seen in FIGS. 3A-3B, therenderer can simulate multiple interactions 300, 310 between the virtualray of light 320 and the volume 120.

To perform path tracing, the renderer takes steps of predeterminedlength along a direction of the virtual ray of light 320. Thepredetermined length can be a millimeter or a centimeter. The renderercan take a step along a virtual ray 325 and reach the point 380. Therenderer can determine whether the point 380 is inside the volume 120 ornot. If the point 380 is outside of the volume, the renderer can tracethe path 390 to the light source to determine the illumination at thepoint 380.

Similarly, the renderer can take a step and reach the point 300. Therenderer can determine whether the point 300 is inside the volume 120 ornot. If the point 300 is inside the volume, the renderer determines theprobability that the ray 320 bounces in a direction 315. The probabilityof the bouncing in direction 315 influences the computation of the colorof the final pixel. For example, the probability can multiply the colorof the ray 320 at the scattering event 300. The final pixel color can bea sum of probabilities multiplied by colors at each of the scatteringevents 300, 305, 310. The renderer takes another step in the direction315, and again determines whether the point 305 is inside or outside thevolume 120. The virtual ray of light 320 can reflect hundreds or eventhousands of times inside the volume 120 before exiting the volume.However, such numerous reflections increase the computation time of thepath tracing simulation.

The renderer can obtain a computational threshold indicating when thesimulation should be terminated. The computational threshold can includea number of interactions 300, 310 between the ray 320 and the volume120. For example, the computational threshold can be 4 or 6interactions.

Upon determining that the number of the multiple interactions exceedsthe computational threshold, the renderer can terminate the simulation.For example, when the number of interactions 300, 310 exceeds fourinteractions, the renderer can terminate the simulation, and it canapproximate interactions between the virtual ray of light 320 and thethree-dimensional object 120 using a second rendering techniquecomputationally less expensive than simulating the multipleinteractions. The second rendering technique can be hundreds of timesfaster than the simulation. The speed of execution of the secondrendering technique cannot exceed a computational threshold, namely, thespeed of execution of simulating the multiple interactions.

The second rendering technique can fall into one of two categories: ascattering approximation or a closed form solution. FIG. 3A shows ascattering approximation. A scattering approximation can include variousrendering techniques such as dipole rendering, multipole rendering, sumof Gaussians, etc. The scattering approximation techniques approximatethe interaction of the ray 320 and the volume 120 without simulatingindividual rays of light scattering through the volume 120.Consequently, the scattering approximations are computationally fasterthan path tracing.

In one embodiment, called “path probing,” when the renderer terminatesthe simulation, the renderer can discard the color contribution of theray 320 that can be computed using path tracing, and instead, therenderer can compute the color contribution of the ray 320 using thescattering approximation. In another embodiment, when the rendererterminates the simulation, the renderer can combine the colorcontribution of the ray 320 that is computed using path tracing and thecolor contribution of the scattering approximation. The scatteringapproximation can be used to determine the color contribution of the ray320 more cheaply than the simulation.

FIG. 3B shows the closed form solution. In the closed form solution,after the computational threshold is reached, for example 6 reflections300, 310, the renderer terminates the simulation and employs a formulato approximate subsequent scattering events. In this case, the renderercombines the color computed using the path tracing and the colorcomputed using the closed form solution, to compute the final color ofthe pixel. For example, the renderer can add the color computed usingthe path tracing and the color computed using the closed form solution,to compute the final color of the pixel.

The formula to approximate subsequent scattering depends on a distance350 between the termination point 330 and an exit point 340 from thevolume 120. The distance 350 is computed in the direction 360 of the ray320 after the last scattering event 370. Further, the formula depends onthe properties of the volume 120 such as absorption coefficient,scattering coefficient, etc. The absorption coefficient indicates howlikely the ray 320 is to be absorbed by the volume 120, while thescattering coefficient indicates how likely the ray 320 is to scatter inthe volume 120. When the ray 320 is absorbed, the ray extinguishes andadds no illumination to the volume 120. When the ray 320 scatters, theray changes direction and continues its path through the volume 120along the changed direction until the next scattering event. Thedistance between the scattering event and the next scattering event isindicated by the scattering coefficient.

FIG. 4 shows a system to present the images with volumetric scatteringto a user. The system 400 to render an image including volumetricscattering includes a software 410, a renderer 420, and a user interface430.

The software 410 can represent a three-dimensional scene includingobject 120 in FIGS. 1 and 3A-3B, and object 220 in FIG. 2 . The software410 can be Maya®, Houdini™, Cinema 4D, Autodesk® 3ds Max®, etc. Thesoftware 410 can pass a representation 440 of the three-dimensionalscene to the renderer 420. The three-dimensional scene can include anindication of which object in the scene should be rendered usingvolumetric scattering.

The processor 450 running the renderer 420 can implement variousrendering techniques 460, 470, 480, including path tracing, scatteringapproximation, and/or closed form solution, as described in thisapplication.

The scattering approximations and the closed form solutions described inFIGS. 3A-3B are but two of the multiple versions of the cheaperrendering techniques. Each version of the cheaper rendering techniquesproduces an image that can be predictably brighter or darker compared tothe image produced using path tracing. The renderer can obtain theinformation a priori about an expected brightness associated with eachcheaper rendering technique, where the expected brightness is measuredwith respect to the path tracing technique. The renderer canautomatically determine how to combine one or more cheaper renderingtechniques based on one or more expected brightness levels.

The renderer can predict the expected brightness of each cheaperrendering technique based on the threshold number of interactions withinthe volume after which the cheaper rendering technique is employed. Forexample, the lower the threshold number, the darker the closed formrendering solution. In another example, the renderer can predict theexpected brightness of the cheaper rendering technique based on size ofthe volume, scattering coefficient of the volume, absorption coefficientof the volume, and the threshold number. For example, if the absorptioncoefficient is high and the scattering coefficient is low, the cheaperrendering techniques can have the same brightness as the simulationbecause the volume is not translucent. In another example, if the sizeof the object is thin, and the scattering coefficient is high, thecheaper rendering techniques can have lower brightness than thesimulation because the volume is translucent.

For example, assume that the expected brightness of the dipoleapproximation is 20% darker than path tracing and the expectedbrightness of sum of Gaussians is 50% brighter than path tracing.Consequently, to obtain the desired brightness, namely, the brightnesscorresponding to the brightness of path tracing, the renderer can weighthe color computed through the dipole approximation and the sum ofGaussians in the following way:

${{pixel}{color}} = {{\frac{5}{7}*\left( {{dipole}{approximation}} \right)} + {\frac{2}{7}*\left( {{sum}{of}{Gaussians}} \right)}}$

The renderer 420 can provide the rendered image 490 to the user throughthe user interface 430. The user can adjust the weighing of the variouscheaper rendering approximations through the user interface 430 and canprovide the new weights 405 to the renderer 420. The user can adjust theweights of the cheaper rendering techniques with or without the renderermaking an initial weight assignment. For example, the user can see theimage 490 rendered using only a single cheaper rendering technique andcan decide to combine two different cheaper rendering techniques.

FIGS. 5A-5C show images obtained using various combinations of thesimulation and cheaper rendering techniques. Rows 500, 510 in FIGS.5A-5C show the images obtained using path tracing along with a closedformula solution. In row 500, the formula includes the out-scattering,which models deflection of the virtual ray of light in differentdirections. The energy of the ray decreases exponentially withincreasing distance.

In row 510 the formula used is the Beer's law. Beer's law can beexpressed as

Energy=e ^((−distance*absorption))

where energy is the energy of the virtual ray of light, distance is thedistance the ray travels through the volume, and absorption is theabsorption coefficient of the volume. As can be seen from the aboveformula, the energy of the virtual ray of light decreases exponentiallywith the distance traveled through the volume.

Row 520 shows the images obtained using the path probing along with adipole rendering technique. The counter 530 shows the number ofinteractions after which the simulation is terminated. The higher thenumber of interactions, the more illumination that comes from pathtracing, but the slower the computation. The variation in the imagesfrom left to right shows increasing translucency of the object 540. Withincreased translucency, the objects are less dense and more see-through,and virtual rays are more likely to find a way through the object.

In FIG. 5A, the threshold number is 0, meaning that no path tracing isperformed. Rows 500, 510 in FIG. 5A show the results of a closed formsolution rendering technique, while row 520 shows the results of dipolerendering technique. As can be seen, the rendering technique shown inrow 500 produces darker results than the rendering technique shown inthe row 510, which produces darker results than the rendering techniqueshown in the row 520. The renderer can combine the various renderingtechniques based on their expected brightness shown in rows 500, 510,520, as described in this application. Brightness can represent anenergy per wavelength.

In FIG. 5B, the threshold number of interactions is five. To produceimages in the rows 500, 510, the renderer combines the color of pathtracing up to five scattering events and the color of two differentclosed form solutions. Consequently, the brightness of the images in therows 500, 510 is increased because a portion of the illumination comesfrom path tracing.

In FIG. 5C, the threshold number of interactions is 60. With 60interactions, most of the illumination is produced by path tracing, andthe illumination in the images in the rows 500, 510, 520 are similar.

FIGS. 6A-6C show areas of an image that are rendered using thesimulation and areas of the images that are rendered using a cheaperrendering technique. In FIGS. 6A-6C, a white color indicates areas ofthe image rendered using the cheaper rendering techniques, such as thedipole approximation, while a pink color indicates areas of the imagerendered using the simulation. The count 600 indicates the thresholdnumber of interactions after which the simulation is terminated. Thevariation in the images from left to right shows increasing translucencyof the object 610.

In FIG. 6A, the threshold number is 0, meaning that no path tracing isperformed. Consequently, the images in FIG. 6A are white. In FIG. 6B,the threshold number is 5, meaning that path tracing is terminated after5 scattering events. As can be seen in FIG. 6B, when the object 610 isnot translucent, and when the threshold number is 5, thin areas 620,630, such as the ears, and high curvature areas 640, 650 are pathtraced. In FIG. 6C, the threshold number is 120, meaning that pathtracing is terminated after 120 scattering events. After 120 scatteringevents, most of the virtual rays exit the object 610, and the color ofthe resulting image is mainly based on path tracing. Consequently, allthe images in FIG. 6C are pink indicating they were rendered using pathtracing.

FIG. 7 is a flowchart of a method to increase a speed of computation ofa volumetric scattering render technique, according to one embodiment.In block 700, hardware or software processor executing instructionsdescribing this application can obtain a threshold number ofinteractions between a virtual ray of light and a three-dimensionalobject through which the virtual ray of light is traveling. Theinteractions can include reflection, absorption or reflection at thesurface of the three-dimensional object. The ray includes one or morewavelengths of electromagnetic radiation. The one or more wavelengthscan be wavelengths in the visible electromagnetic spectrum.

In block 710, the processor can simulate multiple interactions betweenthe virtual ray of light and the three-dimensional object, by, forexample, path tracing the ray of light through the volume occupied bythe three-dimensional object. In block 720, the processor can compare anumber of the multiple interactions to the threshold number ofinteractions.

In block 720, upon determining that the number of the multipleinteractions exceeds the threshold number of interactions, the processorcan terminate the simulation of the multiple interactions. In block 730,the processor can calculate, e.g. approximate, interactions between thevirtual ray of light and the three-dimensional object using a secondrendering technique. The second rendering technique can be moredesirable than the path tracing according to various metrics such as useof computational resources, and/or visual appeal, etc. For example, thesecond rendering technique can be computationally less expensive in CPUconsumption, memory consumption, and/or bandwidth consumption thansimulating the multiple interactions. In another example, the secondrendering technique can produce a more visually appealing result by forexample reducing the noise in the resulting image.

In one embodiment, to approximate the interactions between the virtualray of light and the three-dimensional object the processor can use aformula, as described in this application. Specifically, the processorcan combine the result of the computation from the simulation and theresult of the computation from the approximation to obtain the energycontribution of the ray. Specifically, upon terminating the simulationof the multiple interactions, the processor can compute a first energyper wavelength associated with the virtual ray of light at multiplesample points associated with the three-dimensional object, based on aformula depending on a distance between a termination point and an exitpoint from the three-dimensional object, as described in FIG. 3B. Thefirst energy per wavelength can be the energy of the virtual ray oflight, for a wavelength contained in the virtual ray of light. Themultiple sample points can be points on a voxel grid representing thevolume occupied by the three-dimensional object.

The processor can compute a second energy per wavelength associated withthe virtual ray of light at the multiple sample points associated withthe three-dimensional object, based on simulating the multipleinteractions between the virtual ray of light and the three-dimensionalobject. In other words, the second energy can be computed based on thesimulation.

The processor can combine the first energy and the second energycorresponding to the same wavelength to obtain an energy per wavelengthassociated with the virtual ray of light. Specifically, the processorcan sum the first energy and the second energy. The processor can storethe resulting energy at the multiple sample points associated with thethree-dimensional object.

In another embodiment, to approximate interactions between the virtualray of light and the three-dimensional, upon terminating the simulationof the multiple interactions, the processor can discard a contributioncomputed based on the simulation. Instead, the processor can compute anenergy per wavelength associated with the virtual ray of light atmultiple sample points associated with the three-dimensional objectbased on the second rendering technique to approximate interactionsbetween the virtual ray of light and the three-dimensional object. Thesecond rendering technique can include a dipole, a multiple, a sum ofGaussians, or any other volumetric scattering technique that iscomputationally cheaper than path tracing.

In a third embodiment, to approximate interactions between the virtualray of light and the three-dimensional object, the processor can combinemultiple rendering techniques that are cheaper than the simulation. Theprocessor can obtain multiple rendering techniques computationally lessexpensive than simulating the multiple interactions, such as a dipole,multiple, a sum of Gaussians, etc. The processor can obtain an expectedbrightness associated with each rendering technique among the multiplerendering techniques. The processor can approximate interactions betweenthe virtual ray of light and the three-dimensional object by combiningone or more rendering techniques among the multiple rendering techniquesbased on one or more expected brightness associated with the one or morerendering techniques.

FIG. 8 is a flowchart of a method to increase a speed of computation ofa volumetric scattering render technique, according to anotherembodiment. In block 800, the processor can simulate multipleinteractions between a virtual ray of light and a three-dimensionalobject, as described in this application.

In block 810, the processor can determine whether the simulationincludes a computational threshold. The computational threshold caninclude a total amount of memory used, a total number of CPU cyclesused, and/or a total number of multiple interactions simulated.

In block 820, upon determining that the simulation has exceeded thecomputational threshold, the processor can terminate the simulation ofthe multiple interactions. In block 830, the processor can calculate,e.g. approximate, interactions between the virtual ray of light and thethree-dimensional object using a second rendering technique. The secondrendering technique can be more desirable than the path tracingaccording to various metrics such as use of computational resources,and/or visual appeal, etc. For example, the second rendering techniquecan be computationally less expensive in CPU consumption, memoryconsumption, and/or bandwidth consumption than simulating the multipleinteractions. In another example, the second rendering technique canproduce a more visually appealing result by for example reducing thenoise in the resulting image. The processor can perform additional stepssuch as the ones described in this application, for example in FIG. 7 .

Visual Content Generation System

FIG. 9 illustrates an example visual content generation system 900 asmight be used to generate imagery in the form of still images and/orvideo sequences of images. Visual content generation system 900 mightgenerate imagery of live action scenes, computer generated scenes, or acombination thereof. In a practical system, users are provided withtools that allow them to specify, at high levels and low levels wherenecessary, what is to go into that imagery. For example, a user might bean animation artist and might use visual content generation system 900to capture interaction between two human actors performing live on asound stage and replace one of the human actors with acomputer-generated anthropomorphic non-human being that behaves in waysthat mimic the replaced human actor's movements and mannerisms, and thenadd in a third computer-generated character and background sceneelements that are computer-generated, all in order to tell a desiredstory or generate desired imagery.

Still images that are output by visual content generation system 900might be represented in computer memory as pixel arrays, such as atwo-dimensional array of pixel color values, each associated with apixel having a position in a two-dimensional image array. Pixel colorvalues might be represented by three or more (or fewer) color values perpixel, such as a red value, a green value, and a blue value (e.g., inRGB format). Dimensions of such a two-dimensional array of pixel colorvalues might correspond to a preferred and/or standard display scheme,such as 1920-pixel columns by 1280-pixel rows or 4096-pixel columns by2160-pixel rows, or some other resolution. Images might or might not bestored in a certain structured format, but either way, a desired imagemay be represented as a two-dimensional array of pixel color values. Inanother variation, images are represented by a pair of stereo images forthree-dimensional presentations and in other variations, an imageoutput, or a portion thereof, might represent three-dimensional imageryinstead of just two-dimensional views. In yet other embodiments, pixelvalues are data structures and a pixel value can be associated with apixel and can be a scalar value, a vector, or another data structureassociated with a corresponding pixel. That pixel value might includecolor values, or not, and might include depth values, alpha values,weight values, object identifiers or other pixel value components.

A stored video sequence might include a plurality of images such as thestill images described above, but where each image of the plurality ofimages has a place in a timing sequence and the stored video sequence isarranged so that when each image is displayed in order, at a timeindicated by the timing sequence, the display presents what appears tobe moving and/or changing imagery. In one representation, each image ofthe plurality of images is a video frame having a specified frame numberthat corresponds to an amount of time that would elapse from when avideo sequence begins playing until that specified frame is displayed. Aframe rate might be used to describe how many frames of the stored videosequence are displayed per unit time. Example video sequences mightinclude 24 frames per second (24 FPS), 50 FPS, 140 FPS, or other framerates. In some embodiments, frames are interlaced or otherwise presentedfor display, but for clarity of description, in some examples, it isassumed that a video frame has one specified display time, but othervariations might be contemplated.

One method of creating a video sequence is to simply use a video camerato record a live action scene, i.e., events that physically occur andcan be recorded by a video camera. The events being recorded can beevents to be interpreted as viewed (such as seeing two human actors talkto each other) and/or can include events to be interpreted differentlydue to clever camera operations (such as moving actors about a stage tomake one appear larger than the other despite the actors actually beingof similar build, or using miniature objects with other miniatureobjects so as to be interpreted as a scene containing life-sizedobjects).

Creating video sequences for story-telling or other purposes often callsfor scenes that cannot be created with live actors, such as a talkingtree, an anthropomorphic object, space battles, and the like. Such videosequences might be generated computationally rather than capturing lightfrom live scenes. In some instances, an entirety of a video sequencemight be generated computationally, as in the case of acomputer-animated feature film. In some video sequences, it is desirableto have some computer-generated imagery and some live action, perhapswith some careful merging of the two.

While computer-generated imagery might be creatable by manuallyspecifying each color value for each pixel in each frame, this is likelytoo tedious to be practical. As a result, a creator uses various toolsto specify the imagery at a higher level. As an example, an artist mightspecify the positions in a scene space, such as a three-dimensionalcoordinate system, of objects and/or lighting, as well as a cameraviewpoint, and a camera view plane. From that, a rendering engine couldtake all of those as inputs, and compute each of the pixel color valuesin each of the frames. In another example, an artist specifies positionand movement of an articulated object having some specified texturerather than specifying the color of each pixel representing thatarticulated object in each frame.

In a specific example, a rendering engine performs ray tracing wherein apixel color value is determined by computing which objects lie along aray traced in the scene space from the camera viewpoint through a pointor portion of the camera view plane that corresponds to that pixel. Forexample, a camera view plane might be represented as a rectangle havinga position in the scene space that is divided into a grid correspondingto the pixels of the ultimate image to be generated, and if a raydefined by the camera viewpoint in the scene space and a given pixel inthat grid first intersects a solid, opaque, blue object, that givenpixel is assigned the color blue. Of course, for moderncomputer-generated imagery, determining pixel colors—and therebygenerating imagery—can be more complicated, as there are lightingissues, reflections, interpolations, and other considerations.

As illustrated in FIG. 9 , a live action capture system 902 captures alive scene that plays out on a stage 904. Live action capture system 902is described herein in greater detail, but might include computerprocessing capabilities, image processing capabilities, one or moreprocessors, program code storage for storing program instructionsexecutable by the one or more processors, as well as user input devicesand user output devices, not all of which are shown.

In a specific live action capture system, cameras 906(1) and 906(2)capture the scene, while in some systems, there might be other sensor(s)908 that capture information from the live scene (e.g., infraredcameras, infrared sensors, motion capture (“mo-cap”) detectors, etc.).On stage 904, there might be human actors, animal actors, inanimateobjects, background objects, and possibly an object such as a greenscreen 910 that is designed to be captured in a live scene recording insuch a way that it is easily overlaid with computer-generated imagery.Stage 904 might also contain objects that serve as fiducials, such asfiducials 912(1)-(3), that might be used post-capture to determine wherean object was during capture. A live action scene might be illuminatedby one or more lights, such as an overhead light 914.

During or following the capture of a live action scene, live actioncapture system 902 might output live action footage to a live actionfootage storage 920. A live action processing system 922 might processlive action footage to generate data about that live action footage andstore that data into a live action metadata storage 924. Live actionprocessing system 922 might include computer processing capabilities,image processing capabilities, one or more processors, program codestorage for storing program instructions executable by the one or moreprocessors, as well as user input devices and user output devices, notall of which are shown. Live action processing system 922 might processlive action footage to determine boundaries of objects in a frame ormultiple frames, determine locations of objects in a live action scene,where a camera was relative to some action, distances between movingobjects and fiducials, etc. Where elements have sensors attached to themor are detected, the metadata might include location, color, andintensity of overhead light 914, as that might be useful inpost-processing to match computer-generated lighting on objects that arecomputer-generated and overlaid on the live action footage. Live actionprocessing system 922 might operate autonomously, perhaps based onpredetermined program instructions, to generate and output the liveaction metadata upon receiving and inputting the live action footage.The live action footage can be camera-captured data as well as data fromother sensors.

An animation creation system 930 is another part of visual contentgeneration system 900. Animation creation system 930 might includecomputer processing capabilities, image processing capabilities, one ormore processors, program code storage for storing program instructionsexecutable by the one or more processors, as well as user input devicesand user output devices, not all of which are shown. Animation creationsystem 930 might be used by animation artists, managers, and others tospecify details, perhaps programmatically and/or interactively, ofimagery to be generated. From user input and data from a database orother data source, indicated as a data store 932, animation creationsystem 930 might generate and output data representing objects (e.g., ahorse, a human, a ball, a teapot, a cloud, a light source, a texture,etc.) to an object storage 934, generate and output data representing ascene into a scene description storage 936, and/or generate and outputdata representing animation sequences to an animation sequence storage938.

Scene data might indicate locations of objects and other visualelements, values of their parameters, lighting, camera location, cameraview plane, and other details that a rendering engine 950 might use torender CGI imagery. For example, scene data might include the locationsof several articulated characters, background objects, lighting, etc.specified in a two-dimensional space, three-dimensional space, or otherdimensional space (such as a 2.5-dimensional space, three-quarterdimensions, pseudo-3D spaces, etc.) along with locations of a cameraviewpoint and view place from which to render imagery. For example,scene data might indicate that there is to be a red, fuzzy, talking dogin the right half of a video and a stationary tree in the left half ofthe video, all illuminated by a bright point light source that is aboveand behind the camera viewpoint. In some cases, the camera viewpoint isnot explicit, but can be determined from a viewing frustum. In the caseof imagery that is to be rendered to a rectangular view, the frustumwould be a truncated pyramid. Other shapes for a rendered view arepossible and the camera view plane could be different for differentshapes.

Animation creation system 930 might be interactive, allowing a user toread in animation sequences, scene descriptions, object details, etc.and edit those, possibly returning them to storage to update or replaceexisting data. As an example, an operator might read in objects fromobject storage into a baking processor 942 that would transform thoseobjects into simpler forms and return those to object storage 934 as newor different objects. For example, an operator might read in an objectthat has dozens of specified parameters (movable joints, color options,textures, etc.), select some values for those parameters and then save abaked object that is a simplified object with now fixed values for thoseparameters.

Rather than requiring user specification of each detail of a scene, datafrom data store 932 might be used to drive object presentation. Forexample, if an artist is creating an animation of a spaceship passingover the surface of the Earth, instead of manually drawing or specifyinga coastline, the artist might specify that animation creation system 930is to read data from data store 932 in a file containing coordinates ofEarth coastlines and generate background elements of a scene using thatcoastline data.

Animation sequence data might be in the form of time series of data forcontrol points of an object that has attributes that are controllable.For example, an object might be a humanoid character with limbs andjoints that are movable in manners similar to typical human movements.An artist can specify an animation sequence at a high level, such as“the left hand moves from location (X1, Y1, Z1) to (X2, Y2, Z2) overtime T1 to T2”, at a lower level (e.g., “move the elbow joint 2.5degrees per frame”) or even at a very high level (e.g., “character Ashould move, consistent with the laws of physics that are given for thisscene, from point P1 to point P2 along a specified path”).

Animation sequences in an animated scene might be specified by whathappens in a live action scene. An animation driver generator 944 mightread in live action metadata, such as data representing movements andpositions of body parts of a live actor during a live action scene.Animation driver generator 944 might generate corresponding animationparameters to be stored in animation sequence storage 938 for use inanimating a CGI object. This can be useful where a live action scene ofa human actor is captured while wearing mo-cap fiducials (e.g.,high-contrast markers outside actor clothing, high-visibility paint onactor skin, face, etc.) and the movement of those fiducials isdetermined by live action processing system 922. Animation drivergenerator 944 might convert that movement data into specifications ofhow joints of an articulated CGI character are to move over time.

A rendering engine 950 can read in animation sequences, scenedescriptions, and object details, as well as rendering engine controlinputs, such as a resolution selection and a set of renderingparameters. Resolution selection might be useful for an operator tocontrol a trade-off between speed of rendering and clarity of detail, asspeed might be more important than clarity for a movie maker to testsome interaction or direction, while clarity might be more importantthan speed for a movie maker to generate data that will be used forfinal prints of feature films to be distributed. Rendering engine 950might include computer processing capabilities, image processingcapabilities, one or more processors, program code storage for storingprogram instructions executable by the one or more processors, as wellas user input devices and user output devices, not all of which areshown.

Visual content generation system 900 can also include a merging system960 that merges live footage with animated content. The live footagemight be obtained and input by reading from live action footage storage920 to obtain live action footage, by reading from live action metadatastorage 924 to obtain details such as presumed segmentation in capturedimages segmenting objects in a live action scene from their background(perhaps aided by the fact that green screen 910 was part of the liveaction scene), and by obtaining CGI imagery from rendering engine 950.

A merging system 960 might also read data from rulesets formerging/combining storage 962. A very simple example of a rule in aruleset might be “obtain a full image including a two-dimensional pixelarray from live footage, obtain a full image including a two-dimensionalpixel array from rendering engine 950, and output an image where eachpixel is a corresponding pixel from rendering engine 950 when thecorresponding pixel in the live footage is a specific color of green,otherwise output a pixel value from the corresponding pixel in the livefootage.”

Merging system 960 might include computer processing capabilities, imageprocessing capabilities, one or more processors, program code storagefor storing program instructions executable by the one or moreprocessors, as well as user input devices and user output devices, notall of which are shown. Merging system 960 might operate autonomously,following programming instructions, or might have a user interface orprogrammatic interface over which an operator can control a mergingprocess. In some embodiments, an operator can specify parameter valuesto use in a merging process and/or might specify specific tweaks to bemade to an output of merging system 960, such as modifying boundaries ofsegmented objects, inserting blurs to smooth out imperfections, oradding other effects. Based on its inputs, merging system 960 can outputan image to be stored in a static image storage 970 and/or a sequence ofimages in the form of video to be stored in an animated/combined videostorage 972.

Thus, as described, visual content generation system 900 can be used togenerate video that combines live action with computer-generatedanimation using various components and tools, some of which aredescribed in more detail herein. While visual content generation system900 might be useful for such combinations, with suitable settings, itcan be used for outputting entirely live action footage or entirely CGIsequences. The code may also be provided and/or carried by a transitorycomputer readable medium, e.g., a transmission medium such as in theform of a signal transmitted over a network.

According to one embodiment, the techniques described herein areimplemented by one or more generalized computing systems programmed toperform the techniques pursuant to program instructions in firmware,memory, other storage, or a combination. Special-purpose computingdevices may be used, such as desktop computer systems, portable computersystems, handheld devices, networking devices or any other device thatincorporates hard-wired and/or program logic to implement thetechniques.

One embodiment might include a carrier medium carrying image data orother data having details generated using the methods described herein.The carrier medium can comprise any medium suitable for carrying theimage data or other data, including a storage medium, e.g., solid-statememory, an optical disk or a magnetic disk, or a transient medium, e.g.,a signal carrying the image data such as a signal transmitted over anetwork, a digital signal, a radio frequency signal, an acoustic signal,an optical signal or an electrical signal.

Computer System

FIG. 10 is a block diagram that illustrates a computer system 1000 uponwhich the computer systems of the systems described herein and/or visualcontent generation system 900 (see FIG. 9 ) may be implemented. Computersystem 1000 includes a bus 1002 or other communication mechanism forcommunicating information, and a processor 1004 coupled with bus 1002for processing information. Processor 1004 may be, for example, ageneral-purpose microprocessor.

Computer system 1000 also includes a main memory 1006, such as arandom-access memory (RAM) or other dynamic storage device, coupled tobus 1002 for storing information and instructions to be executed byprocessor 1004. Main memory 1006 may also be used for storing temporaryvariables or other intermediate information during execution ofinstructions to be executed by processor 1004. Such instructions, whenstored in non-transitory storage media accessible to processor 1004,render computer system 1000 into a special-purpose machine that iscustomized to perform the operations specified in the instructions.

Computer system 1000 further includes a read only memory (ROM) 1008 orother static storage device coupled to bus 1002 for storing staticinformation and instructions for processor 1004. A storage device 1010,such as a magnetic disk or optical disk, is provided and coupled to bus1002 for storing information and instructions.

Computer system 1000 may be coupled via bus 1002 to a display 1012, suchas a computer monitor, for displaying information to a computer user. Aninput device 1014, including alphanumeric and other keys, is coupled tobus 1002 for communicating information and command selections toprocessor 1004. Another type of user input device is a cursor control1016, such as a mouse, a trackball, or cursor direction keys forcommunicating direction information and command selections to processor1004 and for controlling cursor movement on display 1012. This inputdevice typically has two degrees of freedom in two axes, a first axis(e.g., x) and a second axis (e.g., y), that allows the device to specifypositions in a plane.

Computer system 1000 may implement the techniques described herein usingcustomized hard-wired logic, one or more ASICs or FPGAs, firmware and/orprogram logic which in combination with the computer system causes orprograms computer system 1000 to be a special-purpose machine. Accordingto one embodiment, the techniques herein are performed by computersystem 1000 in response to processor 1004 executing one or moresequences of one or more instructions contained in main memory 1006.Such instructions may be read into main memory 1006 from another storagemedium, such as storage device 1010. Execution of the sequences ofinstructions contained in main memory 1006 causes processor 1004 toperform the process steps described herein. In alternative embodiments,hard-wired circuitry may be used in place of or in combination withsoftware instructions.

The term “storage media” as used herein refers to any non-transitorymedia that store data and/or instructions that cause a machine tooperation in a specific fashion. Such storage media may includenon-volatile media and/or volatile media. Non-volatile media includes,for example, optical or magnetic disks, such as storage device 1010.Volatile media includes dynamic memory, such as main memory 1006. Commonforms of storage media include, for example, a floppy disk, a flexibledisk, hard disk, solid state drive, magnetic tape, or any other magneticdata storage medium, a CD-ROM, any other optical data storage medium,any physical medium with patterns of holes, a RAM, a PROM, an EPROM, aFLASH-EPROM, NVRAM, any other memory chip or cartridge.

Storage media is distinct from but may be used in conjunction withtransmission media. Transmission media participates in transferringinformation between storage media. For example, transmission mediaincludes coaxial cables, copper wire, and fiber optics, including thewires that include bus 1002. Transmission media can also take the formof acoustic or light waves, such as those generated during radio-waveand infra-red data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to processor 1004 for execution. Forexample, the instructions may initially be carried on a magnetic disk orsolid-state drive of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over anetwork connection. A modem or network interface local to computersystem 1000 can receive the data. Bus 1002 carries the data to mainmemory 1006, from which processor 1004 retrieves and executes theinstructions. The instructions received by main memory 1006 mayoptionally be stored on storage device 1010 either before or afterexecution by processor 1004.

Computer system 1000 also includes a communication interface 1018coupled to bus 1002. Communication interface 1018 provides a two-waydata communication coupling to a network link 1020 that is connected toa local network 1022. For example, communication interface 1018 may be anetwork card, a modem, a cable modem, or a satellite modem to provide adata communication connection to a corresponding type of telephone lineor communications line. Wireless links may also be implemented. In anysuch implementation, communication interface 1018 sends and receiveselectrical, electromagnetic, or optical signals that carry digital datastreams representing various types of information.

Network link 1020 typically provides data communication through one ormore networks to other data devices. For example, network link 1020 mayprovide a connection through local network 1022 to a host computer 1024or to data equipment operated by an Internet Service Provider (ISP)1026. ISP 1026 in turn provides data communication services through theworld-wide packet data communication network now commonly referred to asthe “Internet” 1028. Local network 1022 and Internet 1028 both useelectrical, electromagnetic, or optical signals that carry digital datastreams. The signals through the various networks and the signals onnetwork link 1020 and through communication interface 1018, which carrythe digital data to and from computer system 1000, are example forms oftransmission media.

Computer system 1000 can send messages and receive data, includingprogram code, through the network(s), network link 1020, andcommunication interface 1018. In the Internet example, a server 1030might transmit a requested code for an application program through theInternet 1028, ISP 1026, local network 1022, and communication interface1018. The received code may be executed by processor 1004 as it isreceived, and/or stored in storage device 1010, or other non-volatilestorage for later execution.

Operations of processes described herein can be performed in anysuitable order unless otherwise indicated herein or otherwise clearlycontradicted by context. Processes described herein (or variationsand/or combinations thereof) may be performed under the control of oneor more computer systems configured with executable instructions and maybe implemented as code (e.g., executable instructions, one or morecomputer programs or one or more applications) executing collectively onone or more processors, by hardware or combinations thereof. The codemay be stored on a computer-readable storage medium, for example, in theform of a computer program comprising a plurality of instructionsexecutable by one or more processors. The computer-readable storagemedium may be non-transitory. The code may also be provided carried by atransitory computer readable medium e.g., a transmission medium such asin the form of a signal transmitted over a network.

Conjunctive language, such as phrases of the form “at least one of A, B,and C,” or “at least one of A, B and C,” unless specifically statedotherwise or otherwise clearly contradicted by context, is otherwiseunderstood with the context as used in general to present that an item,term, etc., may be either A or B or C, or any nonempty subset of the setof A and B and C. For instance, in the illustrative example of a sethaving three members, the conjunctive phrases “at least one of A, B, andC” and “at least one of A, B and C” refer to any of the following sets:{A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such conjunctivelanguage is not generally intended to imply that certain embodimentsrequire at least one of A, at least one of B and at least one of C eachto be present.

The use of examples, or exemplary language (e.g., “such as”) providedherein, is intended merely to better illuminate embodiments of theinvention and does not pose a limitation on the scope of the inventionunless otherwise claimed. No language in the specification should beconstrued as indicating any non-claimed element as essential to thepractice of the invention.

In the foregoing specification, embodiments of the invention have beendescribed with reference to numerous specific details that may vary fromimplementation to implementation. The specification and drawings are,accordingly, to be regarded in an illustrative rather than a restrictivesense. The sole and exclusive indicator of the scope of the invention,and what is intended by the applicants to be the scope of the invention,is the literal and equivalent scope of the set of claims that issue fromthis application, in the specific form in which such claims issue,including any subsequent correction.

Further embodiments can be envisioned to one of ordinary skill in theart after reading this disclosure. In other embodiments, combinations orsub-combinations of the above-disclosed invention can be advantageouslymade. The example arrangements of components are shown for purposes ofillustration and combinations, additions, re-arrangements, and the likeare contemplated in alternative embodiments of the present invention.Thus, while the invention has been described with respect to exemplaryembodiments, one skilled in the art will recognize that numerousmodifications are possible.

For example, the processes described herein may be implemented usinghardware components, software components, and/or any combinationthereof. The specification and drawings are, accordingly, to be regardedin an illustrative rather than a restrictive sense. It will, however, beevident that various modifications and changes may be made thereuntowithout departing from the broader spirit and scope of the invention asset forth in the claims and that the invention is intended to cover allmodifications and equivalents within the scope of the following claims.

All references, including publications, patent applications, andpatents, cited herein are hereby incorporated by reference to the sameextent as if each reference were individually and specifically indicatedto be incorporated by reference and were set forth in its entiretyherein.

REMARKS

The terms “example,” “embodiment,” and “implementation” are usedinterchangeably. For example, references to “one example” or “anexample” in the disclosure can be, but not necessarily are, referencesto the same implementation; and, such references mean at least one ofthe implementations. The appearances of the phrase “in one example” arenot necessarily all referring to the same example, nor are separate oralternative examples mutually exclusive of other examples. A feature,structure, or characteristic described in connection with an example canbe included in another example of the disclosure. Moreover, variousfeatures are described which can be exhibited by some examples and notby others. Similarly, various requirements are described which can berequirements for some examples but no other examples.

The terminology used herein should be interpreted in its broadestreasonable manner, even though it is being used in conjunction withcertain specific examples of the invention. The terms used in thedisclosure generally have their ordinary meanings in the relevanttechnical art, within the context of the disclosure, and in the specificcontext where each term is used. A recital of alternative language orsynonyms does not exclude the use of other synonyms. Specialsignificance should not be placed upon whether or not a term iselaborated or discussed herein. The use of highlighting has no influenceon the scope and meaning of a term. Further, it will be appreciated thatthe same thing can be said in more than one way.

Unless the context clearly requires otherwise, throughout thedescription and the claims, the words “comprise,” “comprising,” and thelike are to be construed in an inclusive sense, as opposed to anexclusive or exhaustive sense; that is to say, in the sense of“including, but not limited to.” As used herein, the terms “connected,”“coupled,” or any variant thereof means any connection or coupling,either direct or indirect, between two or more elements; the coupling orconnection between the elements can be physical, logical, or acombination thereof. Additionally, the words “herein,” “above,” “below,”and words of similar import can refer to this application as a whole andnot to any particular portions of this application. Where contextpermits, words in the above Detailed Description using the singular orplural number may also include the plural or singular numberrespectively. The word “or” in reference to a list of two or more itemscovers all of the following interpretations of the word: any of theitems in the list, all of the items in the list, and any combination ofthe items in the list. The term “module” refers broadly to softwarecomponents, firmware components, and/or hardware components.

While specific examples of technology are described above forillustrative purposes, various equivalent modifications are possiblewithin the scope of the invention, as those skilled in the relevant artwill recognize. For example, while processes or blocks are presented ina given order, alternative implementations can perform routines havingsteps, or employ systems having blocks, in a different order, and someprocesses or blocks may be deleted, moved, added, subdivided, combined,and/or modified to provide alternative or sub-combinations. Each ofthese processes or blocks can be implemented in a variety of differentways. Also, while processes or blocks are at times shown as beingperformed in series, these processes or blocks can instead be performedor implemented in parallel or can be performed at different times.Further, any specific numbers noted herein are only examples such thatalternative implementations can employ differing values or ranges.

Details of the disclosed implementations can vary considerably inspecific implementations while still being encompassed by the disclosedteachings. As noted above, particular terminology used when describingfeatures or aspects of the invention should not be taken to imply thatthe terminology is being redefined herein to be restricted to anyspecific characteristics, features, or aspects of the invention withwhich that terminology is associated. In general, the terms used in thefollowing claims should not be construed to limit the invention to thespecific examples disclosed herein, unless the above DetailedDescription explicitly defines such terms. Accordingly, the actual scopeof the invention encompasses not only the disclosed examples but alsoall equivalent ways of practicing or implementing the invention underthe claims. Some alternative implementations can include additionalelements to those implementations described above or include fewerelements.

Any patents, applications, and other references noted above, and anythat may be listed in accompanying filing papers, are incorporatedherein by reference in their entireties, except for any subject matterdisclaimers or disavowals, and except to the extent that theincorporated material is inconsistent with the express disclosureherein, in which case the language in this disclosure controls. Aspectsof the invention can be modified to employ the systems, functions, andconcepts of the various references described above to provide yetfurther implementations of the invention.

To reduce the number of claims, certain implementations are presentedbelow in certain claim forms, but the applicant contemplates variousaspects of an invention in other forms. For example, aspects of a claimcan be recited in a means-plus-function form or in other forms, such asbeing embodied in a computer-readable medium. A claim intended to beinterpreted as a means-plus-function claim will use the words “meansfor.” However, the use of the term “for” in any other context is notintended to invoke a similar interpretation. The applicant reserves theright to pursue such additional claim forms in either this applicationor in a continuing application.

1. A system comprising: at least one hardware processor; and at leastone non-transitory memory storing instructions, which, when executed bythe at least one hardware processor, cause the system to: render a firstimage of a three-dimensional object using a first rendering technique;render a second image of the three-dimensional object using a secondrendering technique; render a third image of the three-dimensionalobject using a third rendering technique, wherein the second renderingtechnique and the third rendering technique are computationally lessexpensive than the first rendering technique; based on the first image,the second image, and the third image, determine a combination includingthe second image and the third image, wherein the combinationapproximates an appearance of the first image; and render multipleimages using the combination including the second rendering techniqueand the third rendering technique, wherein rendering the multiple imagesusing the combination including the second rendering technique and thethird rendering technique is computationally less expensive thanrendering the multiple images using the first rendering technique. 2.The system of claim 1, wherein instructions to determine the combinationincluding the second image and the third image comprise instructions to:obtain a first expected brightness associated with the first image, asecond expected brightness associated with the second image and a thirdexpected brightness associated with the third image; based on the secondexpected brightness and the third expected brightness determine a secondweight associated with the second image and a third weight associatedwith the third image; provide the first image, the second image and thethird image to a user through a user interface; provide the secondweight in the third weight to the user through the user interface;receive a modification to the second weight associated with the secondimage and a modification to the third weight associated with the thirdimage via the user interface; and determine the combination includingthe second image and the third image based on the second weight and thethird weight.
 3. The system of claim 1, wherein instructions todetermine the combination including the second image and the third imagecomprise instructions to: obtain a first expected brightness associatedwith the first image, a second expected brightness associated with thesecond image and a third expected brightness associated with the thirdimage; and automatically determine how to combine the first image andthe second image based on the first expected brightness, the secondexpected brightness, and the third expected brightness, wherein acombination of the second expected brightness and the third expectedbrightness approximates the first expected brightness.
 4. The system ofclaim 1, wherein instructions to determine the combination including thesecond image and the third image comprise instructions to: obtain afirst expected brightness associated with the first image; predict asecond expected brightness associated with the second image and a thirdexpected brightness associated with the third image based on a number ofsimulated interactions between a virtual ray of light and thethree-dimensional object, wherein the lower the number of simulatedinteractions the lower the second expected brightness and the thirdexpected brightness; and automatically determine how to combine thesecond image and the third image based on the first expected brightness,the second expected brightness, and the third expected brightness,wherein a combination of the second expected brightness and the thirdexpected brightness approximates the first expected brightness.
 5. Thesystem of claim 1, wherein instructions to determine the combinationincluding the second image and the third image comprise instructions to:obtain a first expected brightness associated with the first image;predict a second expected brightness associated with the second imageand a third expected brightness associated with the third image based ona size of the three-dimensional object, a scattering coefficient of thethree-dimensional object, and an absorption coefficient of thethree-dimensional object; and automatically determine how to combine thefirst image and the second image based on the first expected brightness,the second expected brightness, and the third expected brightness,wherein a combination of the second expected brightness and the thirdexpected brightness approximates the first expected brightness.
 6. Thesystem of claim 1, wherein instructions to determine the combinationincluding the second image and the third image comprise instructions to:obtain an absorption coefficient associated with the three-dimensionalobject and a scattering coefficient associated with thethree-dimensional object, wherein the absorption coefficient indicates ahigh amount of absorption and the scattering coefficient indicates a lowamount of scattering; determine that a second expected brightness of thesecond image and a third expected brightness of the third image match afirst expected brightness of the first image; and combine the secondimage in the third image using equal weights.
 7. The system of claim 1,wherein instructions to determine the combination including the secondimage and the third image comprise instructions to: obtain a thicknessof the three-dimensional object and a scattering coefficient associatedwith the three-dimensional object, wherein the thickness of thethree-dimensional object indicates a thin object, and the scatteringcoefficient indicates a high amount of scattering; and determine that asecond expected brightness of the second image has lower brightness thana first expected brightness of the first image.
 8. The system of claim1, wherein instructions to determine the combination including thesecond image and the third image comprise instructions to: provide thefirst image, the second image and the third image to a user through auser interface; receive a second weight associated with the second imageand a third weight associated with the third image via the userinterface; and determine the combination including the second image andthe third image based on the second weight and the third weight.
 9. Amethod comprising: rendering a first image of a three-dimensional objectusing a first rendering technique; rendering a second image of thethree-dimensional object using a second rendering technique; rendering athird image of the three-dimensional object using a third renderingtechnique, wherein the second rendering technique and the thirdrendering technique are computationally less expensive than the firstrendering technique; based on the first image, the second image, and thethird image, determining a combination including the second image andthe third image, wherein the combination approximates an appearance ofthe first image; and rendering multiple images using the combinationincluding the second rendering technique and the third renderingtechnique, wherein rendering the multiple images using the combinationincluding the second rendering technique and the third renderingtechnique is computationally less expensive than rendering the multipleimages using the first rendering technique.
 10. The method of claim 9,wherein determining the combination including the second image and thethird image comprises: obtaining a first expected brightness associatedwith the first image, a second expected brightness associated with thesecond image and a third expected brightness associated with the thirdimage; based on the second expected brightness and the third expectedbrightness determining a second weight associated with the second imageand a third weight associated with the third image; providing the firstimage, the second image and the third image to a user through a userinterface; providing the second weight in the third weight to the userthrough the user interface; receiving a modification to the secondweight associated with the second image and a modification to the thirdweight associated with the third image via the user interface; anddetermining the combination including the second image and the thirdimage based on the second weight and the third weight.
 11. The method ofclaim 9, wherein determining the combination including the second imageand the third image comprises: obtaining a first expected brightnessassociated with the first image, a second expected brightness associatedwith the second image and a third expected brightness associated withthe third image; and automatically determining how to combine the firstimage and the second image based on the first expected brightness, thesecond expected brightness, and the third expected brightness, wherein acombination of the second expected brightness and the third expectedbrightness approximates the first expected brightness.
 12. The method ofclaim 9, wherein determining the combination including the second imageand the third image comprises: obtaining a first expected brightnessassociated with the first image; predicting a second expected brightnessassociated with the second image and a third expected brightnessassociated with the third image based on a number of simulatedinteractions between a virtual ray of light and the three-dimensionalobject, wherein the lower the number of simulated interactions the lowerthe second expected brightness and the third expected brightness; andautomatically determining how to combine the second image and the thirdimage based on the first expected brightness, the second expectedbrightness, and the third expected brightness, wherein a combination ofthe second expected brightness and the third expected brightnessapproximates the first expected brightness.
 13. The method of claim 9,wherein determining the combination including the second image and thethird image comprises: obtaining a first expected brightness associatedwith the first image; predicting a second expected brightness associatedwith the second image and a third expected brightness associated withthe third image based on a size of the three-dimensional object, ascattering coefficient of the three-dimensional object, and anabsorption coefficient of the three-dimensional object; andautomatically determining how to combine the second image and the thirdimage based on the first expected brightness, the second expectedbrightness, and the third expected brightness, wherein a combination ofthe second expected brightness and the third expected brightnessapproximates the first expected brightness.
 14. At least onecomputer-readable storage medium, excluding transitory signals andcarrying instructions, which, when executed by at least one dataprocessor of a system, cause the system to: render a first image of athree-dimensional object using a first rendering technique; render asecond image of the three-dimensional object using a second renderingtechnique; render a third image of the three-dimensional object using athird rendering technique, wherein the second rendering technique andthe third rendering technique are computationally less expensive thanthe first rendering technique; based on the first image, the secondimage, and the third image, determine a combination including the secondimage and the third image, wherein the combination approximates anappearance of the first image; and render multiple images using thecombination including the second rendering technique and the thirdrendering technique, wherein rendering the multiple images using thecombination including the second rendering technique and the thirdrendering technique is computationally less expensive than rendering themultiple images using the first rendering technique.
 15. Thecomputer-readable storage medium of claim 14, wherein instructions todetermine the combination including the second image and the third imagecomprise instructions to: obtain a first expected brightness associatedwith the first image, a second expected brightness associated with thesecond image and a third expected brightness associated with the thirdimage; based on the second expected brightness and the third expectedbrightness determine a second weight associated with the second imageand a third weight associated with the third image; provide the firstimage, the second image and the third image to a user through a userinterface; provide the second weight in the third weight to the userthrough the user interface; receive a modification to the second weightassociated with the second image and a modification to the third weightassociated with the third image via the user interface; and determinethe combination including the second image and the third image based onthe second weight and the third weight.
 16. The computer-readablestorage medium of claim 14, wherein instructions to determine thecombination including the second image and the third image compriseinstructions to: obtain a first expected brightness associated with thefirst image, a second expected brightness associated with the secondimage and a third expected brightness associated with the third image;and automatically determine how to combine the first image and thesecond image based on the first expected brightness, the second expectedbrightness, and the third expected brightness, wherein a combination ofthe second expected brightness and the third expected brightnessapproximates the first expected brightness.
 17. The computer-readablestorage medium of claim 14, wherein instructions to determine thecombination including the second image and the third image compriseinstructions to: obtain a first expected brightness associated with thefirst image; predict a second expected brightness associated with thesecond image and a third expected brightness associated with the thirdimage based on a number of simulated interactions between a virtual rayof light and the three-dimensional object, wherein the lower the numberof simulated interactions the lower the second expected brightness andthe third expected brightness; and automatically determine how tocombine the second image and the third image based on the first expectedbrightness, the second expected brightness, and the third expectedbrightness, wherein a combination of the second expected brightness andthe third expected brightness approximates the first expectedbrightness.
 18. The computer-readable storage medium of claim 14,wherein instructions to determine the combination including the secondimage and the third image comprise instructions to: obtain a firstexpected brightness associated with the first image; predict a secondexpected brightness associated with the second image and a thirdexpected brightness associated with the third image based on a size ofthe three-dimensional object, a scattering coefficient of thethree-dimensional object, and an absorption coefficient of thethree-dimensional object; and automatically determine how to combine thesecond image and the third image based on the first expected brightness,the second expected brightness, and the third expected brightness,wherein a combination of the second expected brightness and the thirdexpected brightness approximates the first expected brightness.
 19. Thecomputer-readable storage medium of claim 14, wherein instructions todetermine the combination including the second image and the third imagecomprise instructions to: obtain a thickness of the three-dimensionalobject and a scattering coefficient associated with thethree-dimensional object, wherein the thickness of the three-dimensionalobject indicates a thin object, and the scattering coefficient indicatesa high amount of scattering; and determine that a second expectedbrightness of the second image has lower brightness than a firstexpected brightness of the first image.
 20. The computer-readablestorage medium of claim 14, wherein instructions to determine thecombination including the second image and the third image compriseinstructions to: provide the first image, the second image and the thirdimage to a user through a user interface; receive a second weightassociated with the second image and a third weight associated with thethird image via the user interface; and determine the combinationincluding the second image and the third image based on the secondweight and the third weight.